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ABSTRACT 



This paper describes a mixed method evaluation study that 
was developed to assess faculty teaching behavior change in a faculty 
development fellowship program for community-based hospital faculty. 
Principles of adult learning were taught to faculty participants over the 
fellowship period. These included instruction in teaching methods, group 
facilitation, and curriculum development. Quantitative data were collected 
through the Principles of Adult Learning Scale (PALS) developed by Conti 
(1979) . Qualitative data were also collected through review of videotaped 
teaching sessions. The PALS instrument was administered to 18 faculty in a 
pre- and posttest design, and videotaped sessions of six participants were 
reviewed. Findings of this pilot study indicate that the Fellows' teaching 
beliefs and practices are changing from a teacher-centered to a more 
learner- centered and collaborative mode as a result of the faculty 
development program. This change is reflected in the improvement in their 
overall and individual PALS scores. Video observations also show change in 
the application of adult learning principles in their teaching. The mixed 
method approach provides rich descriptive data about teaching practice. The 
videotapes may not be useful for pure evaluative purposes, but they provide 
valuable feedback for the participants. (Contains two tables and nine 
references . ) (SLD) 
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A. Objectives/Purpose 

The purpose of this session will be to describe a mixed method evaluation study developed to 
assess faculty teaching behavior change in a faculty development fellowship program for 
community-based faculty. Quantitative data for the study was collected by using Conti’s (1979) 
Principles of Adult Learning Scale - PALS instrument. The qualitative data which we compared 
and contrasted with the PALS instrument results were collected via review of videotaped teaching 
sessions presented by program participants during their fellowship year. The qualitative data 
collection was guided by a protocol developed by the authors which seeks to identify and describe 
behaviors congruent with the collaborative mode of teaching defined by seven factors measured in 
the PALS instrument. Our intent was to see how the two data sets might complement each other, 
and how their union might help us to better understand participants’ teaching behavior change. 

The authors will attempt to achieve the following objectives in this session: 

1. Describe a mixed method evaluation approach used to assess faculty teaching behavior change. 

( 

2. Present the PALS as a tool to pre and post test faculty use of the collaborative mode of teaching. 

3. Present the protocol for videotape review and explain how it triangulates the PALS instrument. 

4. Summarize the data from the PALS instrument and the videotapes review. 

5. Use the data to describe how a mixed method evaluation approach may assist faculty developers 
to better understand and assess teaching behavior change. 

6. Explore with participants the benefits and challenges of using mixed methods to evaluate faculty 
change and development. 



Background and Theoretical Framework 

The assessment of change in the teaching of a community-based faculty is a major concern in a 
faculty development program focusing on the integration of adult learning principles into medical 
education. Adult learning principles are taught in our fellowship program through the on site 
mentoring of faculty development specialists and workshops that include the content areas of: 

1. Andragogy: the art and science of helping adults learn (Knowles, 1980) 

2. Introduction to Curriculum Development 

3. Implementing Instruction 

4. Small Group Facilitation 

5. Questioning Skills 





It is essential to assess the participant faculty members’ change in embracing adult learning 
principles, moving from a teacher-centered to learner-centered, collaborative approach in teaching. 
The literature has reported that faculty development program evaluation has been underemphasized 
and that programs must objectively document learning achievement by faculty participants (Skeff et 
al., 1997). Assessment of behavioral change demands a “union of insufficiencies” (Shulman, 

1987, 1988). In other words, multiple methods of evaluation are essential for discerning a 
comprehensive understanding of change. 

Lee Shulman and associates at Stanford have proposed a plan for the assessment of teachers. This 
plan proposes that a full strategy of teacher assessment is a process that demonstrates change and 
accomplishment in instructional development (Shulman, 1987). It is important that this assessment 
not only rely on written tests but also authentically assess the real world performance of teachers. 
Shulman proposes that this assessment should contain the following procedures: “written 
assessments, assessment center exercises, documentation through portfolios, and classroom 
observation.” (Shulman, 1988). 

The community-based hospital faculty development program was developed with many inherent 
parallels to the work of Shulman in emphasizing authentic assessment. The features of evaluation 
in this program are as follows: 1) Pre and post baseline competency evaluations in curriculum 
development, and medical presentation skills are administered to the participants each program 
year. For the 1996-1997 fellowship year we began to use the PALS instrument to assess faculty 
use of the collaborative mode of teaching before and after the faculty development program 
intervention. 2) Faculty Development Specialists spend at least one half day per month with each 
program participant on-site in the community-based hospitals observing teaching activities, 
providing one-on-one feedback to participants and teaching them skills needed for their educational 
encounters. 3) Videotapes of the participants’ teaching activities in the hospital/clinical setting are 
made regularly and feedback is provided on their performance. 4) Participants assemble teaching 
portfolios to document change throughout the program year. 5) Other evaluative instruments such 
exit interviews and focus groups affirm these primary sources and comprehensively aid in the 
triangulation of data. 

As a small part of the data analysis of this larger multifaceted evaluation plan, and taking into 
consideration Schulman’s (1987,1988) notion of a “union of insufficiencies” to discern a 
comprehensive understanding of change, the authors piloted a study which combined quantitative 
(pre and post PALS instrument results) and qualitative (videotape review) data to assess the 
teaching behavior change of 18 faculty who participated in the 1996-1997 faculty development 
fellowship program. Our intent was to see how the two data might complement each other, and 
how their union might help us to better understand participants teaching behavior change. 

Methods 



The study utilized a mixed method approach. First we used a pre and post test research design. 
The PALS instrument was administered to 18 faculty who participated in a community based, part- 
time, year-long faculty development fellowship program. The instrument was administered in the 
beginning and at the end of the fellowship year. 

PALS is a 44 item instrument that measures the frequency with which one practices the 
collaborative teaching mode described in the adult education literature (Conti, 1985). This 
instrument has been used by adult educators in diverse professions to assess a teacher’s acceptance 
and practice of adult learning principles (Conti, 1985). The instrument also measures seven 
individual factors which make up the collaborative mode of teaching: (1) Learner-Centered 
Activities; (2) Personalizing Instruction; (3) Relating to Experience; (4) Assessing Student Needs; 
(5) Climate Building; (6) Participation in Learning Process; and (7) Flexibility for Personal 
Development. This 44 -item instrument is a summative rating scale using a modified Likert scale. 
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Respondents indicate the frequency with which they practice the action described in the items. 
Scores may range from 0 to 220. The mean for the instrument is 146 with a standard deviation of 
20. Scores above 146 indicate a tendency toward a learner-centered approach to the teaching- 
learning transaction, and lower scores imply preference for the teacher-centered approach in which 
authority resides with the teacher. High scores in each factor represent support of the learner- 
centered concept implied in the factor name, and low factor scores indicate support of the opposite 
concept (Conti, 1985,1990). 

After analyzing the results of the PALS, the authors reviewed one pre and one post fellowship 
videotape of participants’ teaching sessions to compare and contrast PALS results with the 
faculty’s real life teaching experiences. The videotape review was guided by a protocol developed 
by the authors which sought to identify and describe behaviors that are congruent with the 
collaborative mode of teaching defined by the seven factors measured in the PALS instrument. 

The protocol was constructed by formulating specific behavior statements out of Conti’s (1990) 
description of the 7 factors of the PALS. This was a difficult task because some of the factors are 
very similar and interrelated, making it difficult to create behavior statements that were specific for 
each factor. We were however, able to come up with items that we thought, were closely related to 
and sometimes spelled out in Conti’s (1990) description of the factors. The protocol has seven 
main behavior statements which represent each of the PALS factors, and under these 7 statements 
are even more specific statements that operationalize the individual factors. Under the 7 behavior 
statements a ranking scale from 1-10 is provided. High scores on each teaching behavior 
statement represent support of the learner-centered concept implied in the PALS factor name. Low 
scores indicate support of the opposite concept. (See appendix A) 

For the purpose of this pilot study, the authors chose to observe the video taped of teaching 
sessions of 6 fellows. We reviewed two tapes for each fellow, pre and post fellowship program 
intervention. We also reviewed 3 others’ video tapes while piloting the protocol, a total of 15 
videos. We chose 6 cases because we decided that twelve video tapes were the amount we could 
handle at this time, since each tape had the duration of 45 min. to 1 1/2 hour. The remaining 
videos will be reviewed in the near future after we carefully train two evaluators. The criteria for 
selecting these 6 cases was the following: we decided to observe the video tapes of the fellows 
who had the highest change of scores between pre and post PALS and those who had the lowest 
change. We thought that these would provide interesting cases to observe since they represent 
both those fellows that made a significant move to a more learner-centered approach of teaching, 
and those that even after the intervention maintained a teacher-centered approach. The authors 
reviewed the videos and scored them separately. We also made sure to write down specific 
vignettes we observed that would support our score for each fellow and each factor under 
observation. After reviewing all the videos we met to discuss our findings and shared with each 
other what we observed. Our scores for each fellow were very close to each other. In most cases 
we kept our original scores for each fellow. We did allow changing of scores for certain factors 
when we were able to remind each other with specific vignettes what took place in the teaching 
sessions that one or the other might have not taken into consideration. Changes were made seven 
out of a possible eighty-four times (8%). This exercise was essential for the qualitative piece of the 
research because it helped us as researchers to better make sense of the phenomenon we were 
attempting to understand, that is, the teaching practice of these fellows. 

PALS Instrument Finding s 

A two-sample hypothesis test was conducted using the pre- and post-test data collected from the 
PALS survey. In General, there was a statistically significant difference between the overall scores 
in the pre-test and post-test surveys. In fact, on average, respondent scores improved by 1 1 .9 
points after the program intervention, a result that was significant at the p = 0.05 level. Comparing 




3 



5 



the performance differences of each respondent in a matched pairs analysis yielded similar results 
(mean = 1 1.97, p = 0.05). 

Of course, the improved performance was not consistent across the seven factors which are 
elements within the PALS instrument. Doing a two-sample hypothesis test of the mean for each 
factor revealed that factors 2 (Personalizing Instruction), 5 (Climate Building), and 6 (Participating 
in Learning Process) showed a statistically significant improvement between the pre-test and the 
post-test results at the standard p = 0.05 level. For factor Personalizing Instruction, the mean 
improvement was 4.8 points; for factor Climate Building, the mean score increased 1.2 points; and 
for factor Participating in Learning Process, the mean score improved by 2.8 points. In addition, 
factor 3 scores increased on average by 1.2 points, but due to increased variance in these 
responses, the difference was only statistically significant at the p = 0. 10 level. These results are 
available in Table 1 below. 



Table 1: Differences in Pre- and Post-Test PALS Scores 



Factor 1 

Meandiff -1.9 

p = >0.10 



Factor 2 Factor 3 
4.8 1.2 

<0.05 <0.10 



Factor 4 Factor 5 
1 1.2 
>0.10 <0.05 



Factor 6 
2.8 
<0.05 



Factor 7 Total 
-0.8 11.9 

>0.10 <0.05 



Additional findings: 

Pre-Test 

• Seventeen fellows (94%) scored below the PALS mean (146). 

• Eleven of these fellows (61%) scored between 1 and 2 standard deviations below the PALS 
mean (Sd=20). 



Post-Test 

• Nine fellows (50%) scored equal or above the PALS mean. 

• Six other fellows’ (33%) scores though improved, fell within one standard deviation below the 
PALS mean. 

• Three fellows (16.6%) decreased their scores between the pre and post-tests. Both of their 
scores were below the PALS mean. One other fellow had a decrease in scores (2.0 points) but 
the post-test score remained above the PALS mean. 

• One fellows’ score though improved, still remained two standard deviations below the PALS 
mean. 



Videotape Finding s 

Twelve video tapes were reviewed from 6 fellows with one pre and one post program intervention 
videotape for each fellow. The ratings between the two evaluators were fairly consistent within a 
two point margin of each other. Overall the video scores were consistent with the fellows’ PALS 
instrument results with the exceptions of Fellows 1 and 2 on video one and Fellow 5 on video two. 
All six fellows scored below the mean (146) on the PALS pre-test which reflects a teacher-centered 
approach to teaching. However, when the evaluators observed video 1 of Fellows 1 and 2, we 
were able to identify behaviors that are congruent with the learner-centered approach to teaching. 
For instance, Fellow 1 administered a needs assessment to identify specific content areas that 
needed to be addressed in the presentation, he provided opportunities for learners to interact by 
developing small group activities, and he established a friendly learning climate, etc. Fellow 2, 
though he utilized a more formal/lecture approach, still chose to utilize other modalities of teaching 
such as small group discussion, open ended questions, and small group tasks. Scoring of the 
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videos for the remaining fellows were consistent with their PALS scores. (See Table 2) 



Table 2 



F ellow 


PALS Pre-test* 


Video 1+ 


PALS Post-test* 


Video 2+ 


1 


124 


54 


150 


49 


2 


118 


49 


147 


58.5 


3 


124 


15.5 


149 


49 


4 


105 


11 


120 


35.5 


5 


123 


14 


120 


15.5 


6 


118 


16 


113 


54 



* 220 points possible. 
+ 70 points possible. 



On video 2 all scores were found to be consistent with the fellows’ PALS post-test with the 
exception of Fellow 6. Fellows 1, 2, and 3 scored above the mean on the PALS post-test, which 
reflects a change from a teacher-centered to a learner-centered approach of teaching. These scores 
were supported by the videotape observations. Fellows 4 and 5 maintained a score below the 
mean in the PALS post test which was congruent with the videotape observations. Though Fellow 
4 had a slight improvement, he still exhibited a propensity for a teacher-centered approach. Fellow 
6 presents a special problem because she had the lowest score in the PALS post-test. However, in 
our observations, we found that she had made significant progress from a teacher-centered to 
learner-centered teaching approach from her first to her second video. It is indiscernible why the 
Fellow’s self assessment on the PALS post test so radically differed from her second observed 
teaching session. 

Discussion 



The finding of this pilot study suggests that Fellows’ teaching beliefs and practice are changing 
from a teacher-centered to a more leamer-centered/collaborative mode, as a result of the faculty 
development program intervention. This change is revealed in the improvement of their overall as 
well as individual PALS scores. The major areas of change focused on three factors: 1) 
Personalizing Instruction (i.e., personalizing learning to meet the unique needs of each learner; 
developing objectives based on the individuals motives and abilities; using various methods, 
materials and learner’s assignments; and promoting cooperation among learners rather than 
competition). 2) Climate Building (i.e., setting a friendly and informal climate for learning; 
encouraging dialogue and interaction in the learning setting; accepting errors as natural part of the 
learning process). 3) Participation in the Learning Process (i. e., allowing learners to identify the 
problems they want to solve and address; allowing learners to participate in making decisions about 
the topics that will be covered in a learning activity; encouraging adult-to-adult relationship between 
teacher and learner) (Conti, 1990). 

Results from our preliminary video observation show the Fellows’ change or lack thereof in their 
application of adult learning principles in their teaching. The qualitative observations have 
provided rich descriptive data that enlighten us about the Fellows’ teaching practice. Secondly, 
observations seem to indicate that overall, the self assessment of the PALS instrument is affirmed 
by the videotape review with a few exceptions. However several limitations still exist in this 
modest study that must be explored further before making definitive conclusions: 

• Researchers have only viewed six fellows’ pre and post teaching experience in addition to three 
pilot videotape reviews. A greater number of videotapes need to be reviewed to further 
develop the protocol and check for consistency in evaluation. Ultimately neutral evaluators 
must be trained in the use of the protocol. 
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• Fellows were not always consistent with the PALS self assessment particularly Fellow 6. The 
researchers need to address why such a large discrepancy might exist between this Fellow’s 
PALS post test and the second videotape evaluation. 

Conclusions 

The mixed method approach provides rich descriptive data concerning fellows teaching practice. 
One may question the need for video tape observation if it only affirms the results of the PALS 
self- assessment. A pragmatic consideration in learner evaluation is that video tape observation is 
an arduous and tedious process. The tapes have validity for fellows’ viewing and conversations 
about teaching process but one may question the need to use them for pure evaluative purposes. 
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Appendix A 

Video Evaluation Form for Conti’s Principles of Adult Learning (PALS) 

1 . The teacher practices learner centered teaching by: 

-Inviting learners to ask as well as answer questions. 

-Leading the learners to find the answer rather than giving the answer. 

-Facilitating discussion rather than simply lecturing. 

1 23456789 10 

Teacher centered Learner centered 

2. The teacher practices personalized instruction by: 

-Giving objectives based on the needs of learners 

-Using applications/illustrations appropriate to needs/levels of learners 

-Promoting cooperation rather than competition 

1 2 3 4 5 6 7 8 9 10 

No personalization Personalized Instruction 

3. The teacher integrates learner experiences by: 

-Referring to previous experiences of learners 
-Making application to learners immediate experience 
-Structuring opportunities for the sharing of learners’ experiences 



1 2 3 4 5 6 

Not relating to experience 


7 


8 9 10 

Relating to Experience 


4. The teacher assesses learners’ needs by: 






-Administrating a needs assessment 
-Checking for understanding 






1 2 3 4 5 6 

Not assessing needs 


7 


8 9 10 

Assessing needs 



5. The teacher builds climate for adult learning by: 



-Addressing learners in friendly and informal manner 
-Moving among the learners instead of staying behind the podium 
-Removing learner risk for making mistakes in discussion 
-Arranging the room to engage learner participation 

123456789 10 

No climate building Climate building 

6. The teacher allows the learner to participate in the learning process by: 

-Encouraging learners to identify the problems that should be addressed 
-Seeking feedback from learners about the content and the teaching process 
-Being flexible to redirect the learning process in response to learners’ problems 

1 2345 6789 10 

No Learner participation Learner participation 

7. The teacher gives flexibility for personal development of the learner by: 

-Serving as a resource person 

-Encouraging learners to talk about feelings and values (ideas and perspectives) 

1 23456789 10 

No flexibility Flexibility 

Developed by Sandro Pinheiro and Jon Rohrer (1998) 
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